1. Field of the Invention
This invention concerns means for inputting characters or commands into a computer or other information receiving device without a keyboard or the like using the automatic skills of handwriting.
2. Description of the Related Art
The present day computer keyboard was initially designed to operate a typewriter. The keys were operated as levers to stamp a die onto paper to print each character. Each key carried two characters one above the other, the lower case character being reproduced by normal depression of a key onto paper with an ink ribbon therebetween and the upper case character being obtained by shifting the entire paper carriage or die set so that the impact occurs with the upper character die impression rather than the lower. Punctuation and special characters were obtained by shifting the numbers or with extra keys.
The printing method is fundamentally the same as in a printing press but the purpose of a typewriter is very different from the purpose of a press. Printing, of course, allows publication of a manuscript and the reproduction of many identical copies of the original manuscript without the effort of handwriting each copy.
The typewriter came into being with the growth of modern commerce and the need for legible business letters. At that time (and indeed presently), handwriting was highly personal and showed great variation. from one person to another. This made handwritten letters, agreements, contracts and other legal documents potentially ambiguous or unclear in meaning. It is this complexity of handwriting which mitigates against current approaches to computer analysis of handwriting.
Variations in handwriting represent simple information embedded in a mass of redundant detail. In modern information and communications, the approach to redundancy in a pattern is to throw large computing power into analysis and recognition. Computer equipment for analysing handwriting is available but does require considerable computing power and hence is relatively expensive and often cannot recognise the handwriting quickly enough, in real time, causing delays to the inputting process.
The analysis employed in such methods depends upon the extraction of salient features from the pattern of handwriting presented to the device and its software. It should be noted that the salient features chosen are often complex and any one may be specific to one character or letter. This implies that the set of such features is large and complex. In addition there exists a number of different ways in which a particular character can be drawn, each of which may contain different salient features. Add to this the difficulty that even with a single way of drawing a particular character, the actual pattern drawn will vary greatly from one person to another. The result is that such approaches to the computer recognition of handwriting have so far been limited in their success and often require a learning process in which the software adjusts to the handwriting of the user or the user learns a way of writing which allows the system to work. The overhead in terms of programme size and computing power required is often expensive and impractical in the application to hand-held computers or personal digital assistants particularly at the smaller end of the scale of size, power and cost (the high volume market of pocket. databanks, diaries, organisers and the like).
Another approach to data input to a computer from finger movements is embodied in systems that require the user to draw each character in a particular way, devoid of ambiguity. This results in a sort of short-hand code which has to be learned by the user. The short-hand forms are often not familiar or readily recognisable as the characters they represent. The result is a commercially successful system but some way removed from natural writing and which needs to be learned and practised.
Another difficulty associated with the current approaches to handwritten input to a computer is the complexity and expense of the hardware required for the sensing of the finger movements. In both the approaches described above, the moment-by-moment and point-by-point form of the motion of the fingers must be sensed, digitised and transmitted to the processor carrying out the analysis and recognition. In many devices currently available this function is performed by a pen or stylus moved by the fingers across a touch sensitive screen. The finger motions are detected by this device and transmitted to the processor, which causes an image of the movement to be displayed on the same screen. Such a complex input device is expensive and can represent a significant proportion of the cost of for example a hand-held computer.
Thus, it is not easy to input hand generated information into a computer in a direct manner.
The printed word, on the other hand, is clear and unambiguous. Every character can be standard in form and scale and easy to read. The printing press sets up its text as a block of lead type which is impressed onto one or more paper pages at a time. This allows the rapid production of many copies of a page. The typewriter, however, needed to be flexible at the level of each character, not at the level of each page. Hence, one key (one print operation) per character. Therefore, the present day keyboard has 60 to 70 keys.
Keyboards which deliver the component parts of each character (one part to one key) have been proposed. Because the form of printed numbers and letters can be simplified (they can be displayed with 7 and 14 segment displays) such a keyboard would only need a relatively small number of keys compared to the standard keyboard. However, such keyboards have not been successful possibly due to the barrier of having to learn a new way of typing which overrides the advantages of such a simple keyboard. It is to be noted that during conventional touch-typing, although the fingers of both hands cover the keys, only one finger is working at a time. With character constructing keyboards as mentioned above, a number of fingers must be employed simultaneously to print a character and so co-ordination skills must be learned by the user. This means that the typing skill called for is less natural than the one-key one-character scheme used by conventional keyboards.
An object of this invention is to provide means for inputting hand generated information into a computer.
According to one aspect of the invention there is provided means for inputting a hand generated character into a computer comprising means for drawing a character, means for abstracting a sequence of signals as the character is drawn corresponding to components of the character to produce a code representative of that character and means for recognising that code, whereby the character is inputted to the computer.
The signal abstracted preferably corresponds to a quantization of motion as the character is drawn. The signal abstracted may correspond to a change in direction as the character is drawn and/or may correspond to movement beyond one or more defined thresholds in a particular direction as the character is drawn and/or a signal abstracted may correspond to a change in position of the drawing means from one defined area to another defined are on a drawing surface.
According to a second aspect of the invention there is provided means for converting movement or force generated in reproducing a character into a coded signal corresponding to one or more elements of said movement or force that are indicative of the character, whereby the character is recognisable from said coded signal.
According to a third aspect of the invention there is provided a device for converting movement of or force applied to at least a part of said device, said movement or force being imparted by reproduction of a character, into a coded signal corresponding to one or more elements of said movement or force that are indicative of the character, whereby the character is recognisable from said coded signal.
According to a fourth aspect of the invention there is provided means for inputting a hand generated character into a computer having a monitor, comprising means for drawing a character to produce a sequence of signals corresponding to that character, means for converting signals produced for one character into a code representative of that character, means for recognising that code and means for providing visual feedback corresponding to the character being inputted as the character is being drawn.
The means according to this aspect of the invention may be used with any handwriting recognition/input system whether involving quantisation recognition or any other system of handwriting analysis.
According to a fifth aspect of the invention there is provided a visual feedback to the writer on a display screen. Feedback may take the form of a sequential build-up or animation of a character form which itself is produced from the above mentioned coded signal. Feedback may be generated by the processor which is connected to the above mentioned input means or input device or any other suitable input means.
Thus the display screen can show the results of the handwriting recognition process as a feedback of information to guide the writer. It preferably operates step by step as the elements of movement are coded by the input device and includes the aspect of computer recognition in the visual feedback process unlike all prior art. It does not indicate the moment by moment movement of the fingers or the point by point form of the character as drawn, as is the case with current approaches to handwriting input to a computer. The user is guided by the interpretation of the finger movements by the system, so as to be able easily and naturally to produce just the correct finger movements that will code as the correct sequence of elements of unambiguous recognition of the writing.
Preferably the visual feedback means comprises means for producing on a monitor a graphic simulation of a character component in response to an abstracted signal. The graphic simulation is preferably modifiable in response to a subsequent signal of a sequence for a character.
The graphic simulation preferably further includes an indicator as to position of the drawing means on a drawing surface. The indicator may comprise an icon displayed at or near an end of the latest graphic simulation component. Alternatively, the indicator may comprise an icon that moves around the graphic simulation of a character in response to movement of the drawing means.
The feedback can be a smoothly produced animation of a cursive character form that responds during its formation to the incoming flow of recognised elements or signal codes.
The computer or input device appears to the user to be cooperating in the process of writing and to be producing the characters on the screen from the prompting provided by the finger movements.
Of course, the characters shown on the screen are not representative of the actual locus or form of the movement of the fingers, but are synthetic representations of the intent of the user, and merely guide the user in the inputting process. From the user""s point of view the characters seem to appear as if written by the user, with the cooperation of the computer.
Such characters can build up to display a completed word, for example, in a standard, clear, joined-up cursive writing, each character of which has been produced from the sequence of simple elements produced by or abstracted from the operation of the input device.
When the user lifts the pen or signals the end of a word in an appropriate. manner, then the processor can immediately replace the cursive characters with the same word displayed in a selected font appropriate to the application or application programme.
In contradistinction to prior art handwriting analysis systems which input information describing the character as drawn and carry out an extraction of salient features (necessarily scale and speed independent), followed by comparison with a stored library of possible shapes, strokes and their inter-relationships, both spatial and temporal, to give the best fit to one character of a complete character set, and thence to the recognised code for the character, the system of the present invention is a direct encoding system where the movements generating the character as drawn, are compared with a single template in such a way that complacent movements directly produce the elements of a code that identifies the character completely by the time the character is completed. At the instant the character is completed, the recognisable code has been completely built and no further analysis or processing is required for recognition.
Preferably recognition occurs character by character in real time. The one or more elements of movement or force are preferably unit vectors.
Preferably analysis of movements or forces into elements is by means of quantizing said movements or forces into one or a sequence of unit vectors. These elements are preferably speed independent, are preferably scale independent and are preferably substantially independent of distortions or variations in the character as reproduced.
Preferably the elements form a set common to all the characters to be reproduced, which set does not contain elements specific to only one or a few characters.
The signal is preferably recognisable by a computer or any other information handling device to which the device is connected, whereby the character can be displayed on a visual display unit operated by the computer or can be processed in the same manner as a character input from a keyboard.
If an input device were activated by movements similar to those employed in writing, then this could provide a method of inputting characters and text into a computer without the need to learn a completely new skill.
What is here described is a device providing a method of analysis which is mechanical or automatic and does not require an indirect process of analysis and comparison to produce a unique code for a character, in contrast to prior art.
This automatic generation of a unique character code may be facilitated by means of a visual feedback from a display of the recognised elements of a character as synthesised from the signal from the input device.
The automatic switch-like method of extracting the coded signal from the finger movements gives rise to relatively simple and inexpensive input devices, recognition contemporaneous with the completion of a handwritten character, low computing power requirement, natural character forms and ease of learning and use, in contrast to prior art.
Thus the invention herein described allows data input to a computer or other system by means of the natural finger movements employed in writing utilising simple and low cost input devices with high speed recognition and visual feedback.
There is an advantage to detecting motion as it is happening as opposed to analysing the space pattern of completed handwriting. The motion of a pen when writing the circle of the letter xe2x80x9caxe2x80x9d is different from the motion when writing the circle of the letter xe2x80x9cpxe2x80x9d, although the resulting shapes are very similar. The xe2x80x9caxe2x80x9d circle is normally produced by an anti-clockwise motion whilst the xe2x80x9cpxe2x80x9d circle is normally made with a clockwise motion. This distinction is lost if the resultant handwritten character is considered after it has been written. However, if the handwriting is analysed dynamically, as it is being written, then the information gained is far more useful. It will be appreciated that references to detection of movement include detection of applied forces in generating said movements.
In a preferred embodiment the drawing means will be a hand held pen or the like, whereby the pen or a part thereof can be moved to reproduce characters.
It is envisaged that the drawing means of the invention will have a part that may be moved relative to a real or notional template when a character is being reproduced and that the drawing means will include means for detecting said movement relative to the template. The template may be incorporated in the drawing means itself or may be separate therefrom. There are various ways in which the movement of said part of the drawing means may be detected.
For example it may be possible to have a template around which said part of the drawing means can be moved, whereby contact of that part of the drawing means on a sensor in a particular part of the template will indicate a direction of movement and again one movement or a sequence of movements will generate a signal corresponding to the character being reproduced by those movements.
Put another way assuming a pen having a body, writing tip and a real template, the template may be separate from the pen, such as on a surface, may be fixed to the pen body or may be fixed to the tip. On the other hand, for a pen having a body and a writing tip, movement of either or both may be relative to a notional template associated with the body, the tip or a separate surface.
The means for detecting a movement of the drawing means or that part thereof may include contact switches, magnetic or capacitive sensors, optical encoders, light detectors, voltage changes, piezo-electric crystal activation or any other suitable means.
The system of the invention preferably includes means for signalling completion of a drawn character. Completion may be signalled by lifting the drawing means from a drawing surface. Alternatively, completion of a character may be indicated by a unique movement of the drawing means relative to that character. Another alternative may be to indicate completion of a character by movement of one of the drawing means and an icon indicative of the drawing means to a defined position, possibly on the drawing surface or an area defined on a monitor.
The mode of analysis envisaged by the invention is actually concerned with the time patterns of muscle action, in contrast to the space patterns of completed handwriting. It is relevant to note that all communication occurs through the medium of muscle action, whether speech, body language, touch, action, handwriting or typing. The first outward expression of thought is always through muscular action. This invention is aimed at allowing the communication with a computer to take place at the level of the neuromuscular skill of writing.
It will be appreciated, however, that there is considerable redundancy present in handwriting. Although handwriting may be taught in a uniform fashion, variations and embellishments are added as a person develops his handwriting skill, so that whilst letters and words can be recognised, it is extremely difficult for, for example, a computer scanning device to extract the essential characters because of personal variations and embellishments.
Accordingly, a preferred aim of the device of the invention is to enable characters to be reproduced as unit vectors. In other words, each character as it is drawn using the device of the invention preferably produces a signal for that character as one or a sequence of steps. This may be achieved by limiting or restricting registration of the movement to one or a series of quantized steps or unit vectors.
It is important to realise that signals which solely describe the position, movement or locus of a pen or moving part of the device simply provide a copy of that movement etc. in electronic, electrical etc form. They do not of themselves facilitate logical recognition of the inputted letter form or character form.
What this invention allows is an automatic reduction of the movement etc into a quantized form. This means that the movement is divided into steps which indicate the time sequence of unit vectors which characterise the movement etc. The steps themselves do not describe the point by point and moment by moment movement which results from drawing the character form. They are rather the result of an analysis of the movement etc which indicates a series of unit vectors. This series of unit vectors cannot be used to reconstruct the original finger movements, because all redundant space and time information is discarded in the process of detecting the unit vector sequence. All that remains is the sequence of the unit vectors and the character of the unit vectors.
The character of the unit vectors will be dependent on the design of the device. In the case of a physical square template the unit vectors could be characterised for example as being up, down, left or right.
The time delay between one unit vector and the next is not of importance and is discarded information. All that matters for recognition is the sequence, eg. left then down then right then up then down for the handwritten letter form xe2x80x9caxe2x80x9d.
Also the process of deriving the unit vectors disregards the scale or size of the movement or letter form. The same sequence of the same unit vectors results from a large xe2x80x9caxe2x80x9d as from a small xe2x80x9caxe2x80x9d. In addition, provided the physical movements which activate the movement or position detectors are smaller than the smallest character to be drawn, the sequence of unit vectors will be the same for wide variations or distortions in the form of the original character, letter or resulting motion.
It should be noted that such a family of unit vectors (one simple case being: UP, DOWN, LEFT, RIGHT) can represent all the characters to he input to a computer etc through finger movements.
In other words, each and every number, letter etc can be analysed into a sequence of the same set or family of unit vectors. The uniqueness of character resides in the sequence of the unit vectors which represents a unique code for the character. The different characters do not require analysis into unique individual features as in the prior art.
The analysis of original motion into unit vectors is according to a scheme which compares the movement to an arrangement of detectors placed in a fixed relation to a real or notional template. This allows the motion to be compared with the geometry of a template in such a way that a complacent movement will result in a single signal or part of a signal which indicates the characteristic direction or movement at that stage of the drawing of the letter or character etc.
For example, once the moving part has gone beyond the upper limit of detection, the unit vector will indicate simply xe2x80x9cupxe2x80x9d until the moving part has once again returned within the scope of detection in the direction, when it could be followed by xe2x80x9cdownxe2x80x9d. Similarly with horizontal movement. This approach leads naturally to a description of operation of the device in terms of a template.
The template is simply the geometry which determines the signalling of the unit vectors, and may be a physical form eg. a square aperture within which the pen tip etc. moves, or it may be notional, and is simply the space pattern of detector switching limits in two dimensions or it may be embodied in the movement analysing processor which is connected to the input device moved by the fingers.
Either scheme will result in practical devices which convert the finger and hand movement familiar to us as handwriting into a code signals which is logically recognisable as corresponding to the character drawn.
For accuracy of coding, and in order to remove the inaccuracies introduced by personal embellishments, the writer may be guided by visual feedback from an image on a display screen, and can choose natural character shapes which can be learned quickly and easily.
Thus the device allows xe2x80x9ctypingxe2x80x9d or inputting or textual information into a computer or other automatic text handler (eg. typewriter, portable databank or diary etc.) at handwriting speeds or faster, without the need to learn the far more complicated skills of touch typing using a conventional keyboard.
The principle of operation is based on the quantization of motion, and is not to be confused with handwriting analysis which causes automatic recognition of the form of normal personal handwriting (or even the recognition of a limited or defined or stylised set of character forms) by an analysis of its complex actual shape.
The aim of the template either real or notional is to register the movement of the device as unit vectors but not necessarily to restrict the movement of the device to unit vector form, whereby a recognisable signal corresponding to that character can be produced.
In preferred forms of the invention the relation between the template and the part or parts of the device will be flexible, thereby freeing the device from performing forced angular, rectangular or linear movements. In other words, by introducing a flexible linkage between relatively movable parts of the device or between a movable part of the device and the template, the device can follow both straight and curved lines whilst those movements will be detected as straight line movements or forces producing unit vectors.
Thus, the preferred device of the invention has the ability to detect movements of at least a part thereof in producing a character as one or a sequence of unit vectors to produce a signal corresponding to the character, even when the character is not reproduced in a format constrained by the geometry of the template.
The flexible linkage may take any suitable form. For example, when the tip of a pen device is to be movable relative to the body of the device, the flexible linkage may be provided by one or more elastic members linking the tip to the body.
Various considerations may be taken into account in deciding the nature of the real or notional template.
In one preferred embodiment, the template may be in the form of an enclosure having at spaced positions around its periphery means for detecting movement of said device part from one point to another around the periphery of the enclosure. The enclosure may be of any suitable shape but will preferably be a square or a circle. Preferably four detection positions will be provided at equidistant spacings.
The movable part of the device may be a rod or the like and its movement from one detection point to another may be by any suitable sensor means, such as already suggested above.
In another preferred embodiment, the template may be in the form of a confined track around which the movable part of the device can travel, again with spaced detection points as in the first preferred embodiment.
In a yet further preferred embodiment, the template is notional rather than real and may be embodied in the processor running the requisite software and the movable part of the device may be detectable as being in accordance with a template. Thus, the device of this preferred embodiment of the invention will include means for registering the movement of said movable part as though it were following a template. Thus, the device may be arranged to produce output signals when movement of at least a part thereof exceeds a notional boundary of the notional template.
It will be appreciated that these signals indicate major changes in direction as compared to a template or set of directions or axes. It is possible to derive the signals indicating the unit vectors as changes in velocity or other time derivatives as well as direction or position. Such a derivation is suited to the application of this invention to conventional computer pointing equipment.
For example the data stream from a computer pointing device such as a mouse, trackball, pen and tablet etc indicates the relative position of the fingers moment by moment. If this data stream is analysed by a computer or dedicated processor in such a manner that excursions of the finger position are compared with a notional template, encoded in an algorithm stored within the computer or processor or its associated memory as a pattern of excursion limits in two dimensions, movements beyond these limits or complacent with the template boundaries can trigger the generation of a sequence of signals, indicative of the unit vectors, which codes uniquely for the character drawn by the fingers which are moving the mouse, trackball, pen and tablet or other pointing device.
This invention will now be further described, by way of example only, with reference to the accompanying drawings, in which: